Evaluation of the Allplex HPV assay’s adherence to international guidelines for cervical cancer screening in clinician-collected samples

ABSTRACT Clinically validated human papillomavirus (HPV) assays are crucial in cervical cancer screening. In this study, we evaluated the Allplex HPV HR Detection assay (Seegene, SouthKorea) for its clinical accuracy and reproducibility according to the international criteria, using the RealTime High Risk HPV m2000 assay (Abbott, USA) as standard comparator. The Allplex HPV HR assay exhibits significant non-inferior sensitivity to detect cervical intraepithelial neoplasia grade (CIN) 2 or worse (CIN2+) with a ratio of 1.00 (95% CI: 0.97–1.03, P = 0.006), insignificant non-inferior sensitivity to detect CIN3+ with a ratio of 1.00 (95% CI: 0.88–1.13, P = 0.098), and non-inferior specificity to exclude CIN2+ with a ratio of 0.99 (95% CI: 0.99–1.00, P < 0.001) compared to the standard comparator. In addition, the assay shows an excellent reproducibility within the same laboratory [96.5% (95% CI: 94.6–97.9) with a kappa value of 0.91 (95% CI: 0.87–0.95)] and between laboratories [96.7% (95% CI: 94.8–98.0) with a kappa value of 0.91 (95% CI: 0.87–0.95)] for overall high-risk HPV positivity as well as for each individual HPV type. Pooling our study data with those of another independent study supports the consistency of our findings. We conclude that both the clinical accuracy to detect cervical precancer and the reproducibility of Allplex HPV HR Detection assay fulfill the international validation criteria of use in cervical cancer screening. IMPORTANCE The clinical validation of human papillomavirus (HPV) assays in accordance with well-established international guidelines is crucial to ensure that only validated assays are used in the context of screening (Meijer et al., Int J Cancer, 2009). The guidelines, developed by an international consortium, require that a novel HPV assay has non-inferior accuracy against a standard comparator test for the detection of cervical intraepithelial neoplasia grade (CIN) 2 or worse (CIN2+). Additionally, a new HPV assay should meet specific criteria for both intra- and inter-laboratory reproducibility to ensure the assay consistently exhibits technical precision and robust performance. Pooling our study data with those of another independent study supports the consistency of our findings. In conclusion, both the clinical accuracy to detect cervical precancer and the reproducibility of Allplex HPV HR Detection assay fulfill the international validation criteria of use in cervical cancer screening.

infection of the reproductive tract, 90% of infections resolve spontaneously.However, persistent HPV infection carries a risk of progression to CC with the disease typically evolving over an extended timeframe, taking 15-20 years for CC to develop.The slow disease progression consequently makes screening an effective public health interven tion that enables timely detection and early intervention.Despite an increasing global effort to promote HPV vaccination among young cohorts, CC screening continues to be a cornerstone of public health initiative in alleviating the CC burden in women aged 30 and above.
Evidence from large-scale randomized clinical trials clearly shows that hrHPV-based screening significantly reduces the incidence and mortality of CC compared to traditional cytology-based screening (5)(6)(7).The World Health Organization currently recommends HPV nucleic acid amplification tests as a first-choice screening method for CC (8).A 2020 review revealed over 250 HPV assays available on the market with over 90% of the assays lacking adequate evaluation and clinical validation (9).
Clinical validation of HPV assays in accordance with well-established international guidelines is crucial to ensure that only validated assays are used in the screening context (10).The guidelines, developed by an international consortium, require that a novel HPV assay has non-inferior accuracy against a standard comparator test (SCT) for the detection of cervical intraepithelial neoplasia grade (CIN) 2 or worse (CIN2+).Additionally, a new HPV assay should meet specific criteria for both intra-and inter-labo ratory reproducibility to ensure the assay consistently exhibits technical precision and robust performance.
The Allplex HPV HR Detection assay (Seegene, Seoul, South Korea) (referred as "AllplexHPVHR") is a novel multiplex real-time fluorescence polymerase chain reaction (PCR) assay that detects 14 hrHPV types.This study set out to evaluate the clinical accuracy and reproducibility of AllplexHPVHR, with cervical cell samples originating from primary CC screening in Belgium.Recently, Oštrbenk et al. (11) reported a clinical accuracy and reproducibility study with AllplexHPVHR in a Slovenian CC screening population, using a different comparator [i.e.,Hybrid Capture 2 (HC2, Qiagen, USA)].We have extracted data from this paper and performed a meta-analysis on the relative accuracy of AllplexHPVHR.

Study design and study population
The Meijer guidelines are a study protocol to validate HPV assays for the use in CC screening (10).According to this guideline, an HPV assay should fulfill the accuracy and reproducibility (intra-and inter-laboratory) criteria using samples that are represen tative for a screening population.In our study, we used remnant material of cervical cells from women participating in primary CC screening in Belgium.The clinician-collec ted samples were stored according to the manufacturer's instructions in liquid-based cytology medium (PreservCyt, Hologic, Marlborough, MA, USA) at the Sonic Healthcare Benelux (AML) biobank.
For the clinical accuracy assessment, 860 samples were included in a panel (referred as "AML_Meijer_1") of 60 CIN2+ cases (of which 16 cases are confirmed CIN3+) and 800 cytological and viral negative controls; all selected according to the Meijer criteria (10) and from a population-based screening cohort.The 60 CIN2+ samples were randomly selected out of a pool of cervical specimens from women with a histologically confirmed CIN2+ result.HPV positivity was determined using the Riatol qPCR assay (12) as a pre-selection.Viral loads were not considered for the selection of the 60 CIN2+ histologypositive samples, only Ct-values were documented.Controls were randomly selected out of a pool of cervical specimens from women of at least 30 years of age.These women had a documented negative HPV test result (12) and a prior negative cytology screening result no more than 3 years before.
To assess the intra-and inter-laboratory reproducibility, 550 screening samples were selected from the AML biobank for the reproducibility panel (referred as "AML_Repro_1").The 550 samples were tested using the Abbott RealTime High Risk HPV m2000 assay (Abbott, Chicago, IL, USA) (referred as "RealTime").The AML_Repro_1 panel includes 176 RealTime-positive samples and 374 RealTime-negative samples, according to validation guidelines of including at least 30% HPV-positive samples (10).

Index test
The index test was the Seegene AllplexHPVHR assay, an innovative qualitative realtime PCR-based assay that incorporates patented technologies such as Dual-Priming Oligonucleotides (DPO), Tagging Oligonucleotide Cleavage and Extension (TOCE), Multiple Detection Temperature (MuDT), and "3 Ct" to allow simultaneous detection of multiple targets per fluorophore.AllplexHPVHR targets the E6/E7 region for DNA detection of HPV 16

Standard comparator test
The SCT was the Abbott RealTime High Risk HPV m2000 assay, which is a second-gen eration standard comparator (communicated at EUROGIN-International Multidiscipli nary HPV conferences in 2023 and 2024 and at the 35th International Papillomavirus Conference 2023).Since the standard comparator hrHPV tests (HC2 and GP5+/6+) are infrequently used in current clinical practice and lack genotyping capabilities, newer hrHPV DNA comparator tests have been proposed, and RealTime has been added to the list of alternative hrHPV DNA comparator tests (13).RealTime is an automated multiplex PCR, targeting the L1 gene.It detects 14 hrHPV genotypes and has been consistently validated in several previous studies (14)(15)(16).RealTime allows separate genotyping of HPV 16 and HPV 18 and an aggregate of 12 other hrHPV genotypes (HPV 31,33,35,39,45,51,52,56,58,59, 66, and 68) (15).

Clinical accuracy testing
Accuracy criteria testing was carried out at AML, and the panel was tested once with the SCT and once with the index test according to the manufacturer's recommendations.Briefly, all primary samples from the AML_Meijer_1 panel were extracted in batches of 92 samples using the Seegene STARMag96 Universal Extraction System on the fully automated Seegene STARlet instrument according to the manufacturer's instructions for AllplexHPVHR.For PCR amplification, a Bio-Rad CFX96 instrument was used in combi nation with the Seegene Viewer (V3.30.000, trial version) for subsequent HPV genotyp ing interpretation and clinical cut-off implementation.According to the manufacturer, other compatible DNA extraction and/or PCR instruments-not used in this study-are SEEPREP32 (an extraction-only system), Seegene STARlet-AIO (an all-in-one automated extraction and PCR system), Seegene STARlet 96MPH, and Seegene NIMBUS (the latter two are automated extraction and PCR setup systems).In parallel, the SCT was per formed on the same primary samples, which were first extracted with the Abbott mSample Preparation System on the fully automated Abbott m2000sp instrument.PCR amplification was performed on the Abbott m2000rt instrument, and results were compared to the AllplexHPVHR results.

Reproducibility testing
To assess the AllplexHPVHR reproducibility, three aliquots of remnant primary material were obtained from each sample where aliquots 1 and 2 were tested with AllplexHPVHR within the same laboratory (AML) for the intra-laboratory assessment (run 1 versus run 2).Run 1 was carried out on 3 and 4 April 2023 and run 2 on 5 and 6 April 2023.The third aliquot was sent to another laboratory (Laboratory of Clinical Biology, UZ Ghent) for inter-laboratory reproducibility (assessed between 3 and 7 April 2023); whereafter, the results were compared to the run 1 results obtained from AML.Briefly, all primary samples from the AML_Repro_1 panel followed the same sample handling procedure for AllplexHPVHR at AML as with the AML_Meijer_1 panel.This procedure was carried out twice at AML for the intra-laboratory assessment.At UZ Ghent, samples were maintained at 4°C until processing, involving a brief spin and the transfer of 300 µL into an extraction tube (comprising 100 µL dead volume).Nucleotide extraction was executed using the STARMag96 × 4 Universal Cartridge Kit, with an elution volume of 100 µL.Subsequently, 5 µL of DNA extract was employed in the AllplexHPVHR protocol, utilizing default parameters.

Statistical analyses
Clinical performance of AllplexHPVHR was evaluated in terms of absolute sensitivity for CIN2+ and specificity for controls, with 95% CI.When the +/− and −/+ results were zero, CIs were computed by adding 0.5 to each cell of the contingency table.To determine the non-inferiority of the index test compared to SCT, we followed Tang's proposed method, which uses benchmarks of 0.90 for relative sensitivity and 0.98 for relative specificity.
Reproducibility was measured as the overall percentage agreement, which indicates the proportion of concordant results (i.e., both tests being positive or both negative) out of all results.The reproducibility criterion was considered met if the lower bound of the 95% CI for concordance of all hrHPV types and of each hrHPV type separately exceeded 87%, and if the kappa (κ) statistic was >0.5.
Meta-analysis of relative sensitivity and specificity of AllplexHPVHR to comparator was performed using a random effects model for pooling ratios of proportions, and a forest plot was constructed (17).Non-inferiority was confirmed when the left 90% CI bound around the relative accuracy exceeds 0.90 (for sensitivity) or 0.98 (for specificity).
Furthermore, 95% exact CIs were computed for all proportions, considering the paired design in comparisons when calculating confidence intervals for relative sensitivity and specificity.Statistical analyses were conducted using STATA version 16 (College Station, TX, USA).

Clinical performance
The internal controls of both AllplexHPVHR and RealTime were amplified yielding 860 valid samples for the clinical accuracy analyses.For the clinical sensitivity analysis, AllplexHPVHR yielded a positive result for 56/60 CIN2+ cases, resulting in an absolute clinical sensitivity of 93.3% (95% CI: 87.0-99.6).RealTime also showed 56/60 positive results for CIN2+ showing identical results of absolute clinical sensitivity.The relative sensitivity of AllplexHPVHR was 1.00 with a P-value for non-inferiority (P ni ) of 0.006, indicating non-inferior clinical sensitivity of AllplexHPVHR compared to RealTime.For the identification of CIN3 cases, both assays detected 15/16 CIN3 cases and missed 1 CIN3 case.For controls, AllplexHPVHR yielded negative results for 795/800 samples [absolute specificity: 99.4% (95% CI: 98.8-99.9)],whereas RealTime was negative for 798/800 controls [absolute specificity: 99.8% (95% CI: 99.4-100.0)].AllplexHPVHR was non-inferior for specificity compared to RealTime (P ni < 0.001).The clinical performance estimates are shown in Table 1.Out of the 56 concordantly positive CIN2+ cases detected by both AllplexHPVHR and RealTime, 14 were single HPV 16 infections, 3 were single HPV 18 infections, 26 were other infections not containing HPV 16 or HPV 18, and 13 were multiple infections (Table 2).In Table 3, we describe the prevalence of type-specific HPV infections, determined by AllplexHPVHR among CIN2+ cases in two rankings: (i) non-hierarchical and (ii) hierarchical.The hierarchical ranking is according to the risk of cervical cancer associated with each HPV type (3).HPV 16, the most potent carcinogenic type, was the most frequently detected among the CIN2+ cases, accounting for 44.6% (25/56).HPV 18, the second most potent carcinogenic type, was found in seven cases (12.5%), with two of these cases co-infected with HPV 16.HPV 16 or 18 was collectively present in 30 cases (53.6%).Additionally, in 50 cases (89.3%), one or more HPV types included in the nonavalent vaccine were detected (Table 3).
Additionally, the reproducibility of AllplexHPVHR at the genotype-specific levelboth in the intra-and inter-laboratory analysis-consistently demonstrated an overall agreement between 97% and 99% for HPV 16 and the 10 other hrHPV types and 100% for HPV 18, 35, and 58 (Tables 5 and 6).The kappa values for all the specific genotypes were higher than 0.80.The intra-laboratory general agreement of the hrHPV types separately ranges from 99.3% to 100.0% (κ = 0.83-1.00),indicating an excellent intra-laboratory reproducibility of each hrHPV type (Table 5).Similar results are found in the inter-laboratory general agreement (99.3%-100.0%)and the kappa value of each hrHPV type (0.80-1.00) (Table 6).

DISCUSSION
In this comprehensive validation study, AllplexHPVHR demonstrated non-inferior clinical sensitivity and specificity for the detection of cervical precancer CIN2+ compared to the SCT, and the non-inferiority for the identification of CIN3+ cases compared to the SCT was not assessed due to low number of CIN3+.Additionally, AllplexHPVHR exhibited excellent intra-and inter-laboratory reproducibility for overall results of the assay and for each hrHPV type separately.This study establishes the AllplexHPVHR assay as the fifth full genotyping HPV assay that is fully validated.AllplexHPVHR makes use of several patented technologies from Seegene, including (i) DPO, (ii) TOCE that includes a pitcher and a catcher in the reaction, (iii) MuDT, and (iv) "3 Ct" technology.The TOCE approach makes AllplexHPVHR unique in distinguishing HPV types separately in one reaction tube, unlike a conventional real-time PCR having a set of primers and probe.The MuDT technology allows simultaneous detection of various targets in the same fluorescent channel and at multiple pre-programmed temperatures,    c Hierarchical: Samples with a single and a multiple infection of the most carcinogenic HPV type (HPV 16) are counted first, followed by adding samples that contain single infection with the second-most carcinogenic HPV type (HPV 18) on top of HPV 16 (single and multiple infection) and so forth (in an ascending way).
unlike a melting temperature analysis where the temperature decreases by a gradient while reading the fluorescent signal.AllplexHPVHR is a low-cost and relatively highthroughput assay.Although the procedure of the automated STARlet system is quite complex, the integrated step-by-step real-time instructions provided by the system are user-friendly.The automated workflow from DNA extraction until obtaining PCR results takes about 4 hours.No random access is allowed, and samples can only be analyzed in batch.In an 8-hour shift, 3 runs of 92 samples can be performed with AllplexHPVHR.Similar performance is witnessed with RealTime.However, AllplexHPVHR detects 14 hrHPV types separately and simultaneously; whereas RealTime detects HPV 16, HPV 18, and the other 12 hrHPV types as aggregate.AllplexHPVHR and the STARlet system are more suitable for high-throughput laboratories than low-throughput laboratories.Maintenance and troubleshooting of the automated STARlet system require expertise and resources, which can be less suitable for low-resource setting.Oštrbenk et al. concluded from a Slovenian study, similar to ours, that the interna tional criteria for HPV assay validation of AllplexHPVHR were fulfilled (11).Oštrbenk et al. included 110 cases (52 CIN2, 55 CIN3, and 3 squamous cell cervical cancers) and 863 controls defined by screening with cytology and HPV assay, and a previously  negative cytology screening results 3 years before.Our study independently confirmed the Slovenian findings, and the meta-analysis highlighted consistency of both validation studies.We note that there is a slight difference in the relative specificity between this study and the Slovenian study.One of the possible explanations is the selection criteria of the controls: in this study, the controls were selected based on a negative HPV test result with a prior negative cytology screening result of no more than 3 years before, while in the previous studies, the controls selection was based on cytology test results at the time of screening and the prior screening (16,18,19).Another possible explanation is the usage of different standard comparators: in the Slovenian study, the HC2 assay which is a signal amplification assay known to cross-react with HPV types not belonging to the IARC group I carcinogenic types, while the PCR-based Abbott RealTime assay was used in this study.
As far as we know, the Abbott RealTime HPV test is used for the first time as a stand-alone second-generation standard comparator in a clinical assessment.There are four criteria for an HPV assay to be fulfilled in becoming a second-generation SCT: (i) consistent validation of the clinical accuracy [sensitivity for both CIN2+ and CIN3+ and specificity for controls in at least three other studies against the first-generation SCTs (which includes HC2 and GP5+/6+ PCR EIA assays)]; (ii) pooled non-inferior sensitivity for CIN2+ and CIN3+ using 0.95 as benchmark compared to the first-generation SCTs; (iii) documented intra-and inter-laboratory reproducibility in at least one validation study; and (iv) target all the 12 IARC group I carcinogenic HPV types (i.e., HPV 16,18,31,33,35,39,45,51,52,56,58, and 59) with allowance for HPV 66 and HPV 68 (since also targeted by the first-generation SCTs).The RealTime assay was validated against HC2 in four studies (14,15,20,21) and against GP5+/6+ PCR EIA in one study (16).
Limitations of our study include the study design that may be prone to selection bias in cases and controls.Despite this, the Meijer protocol is a well-established protocol and is accepted by the international scientific community as a methodology sound design to validate HPV assays.Designs such as the VALGENT protocol have been developed, where validation of HPV assays is carried out by using samples integrated into popula tion-based, well-established screening programs, bolstered by comprehensive registries.Additionally, there is a limitation in the specificity assessment since the HPV-positive rate in the controls is lower (i.e., 0.62%) than in previous similar studies (16,18,19).This is explained by the selection criteria of the controls.Another limitation is that the AllplexHPVHR is a full genotyping assay, while RealTime is a partial genotyping assay that identifies HPV 16, HPV 18, and the other aggregated hrHPV types.This does not allow the comparison of the type-specific performance.So far, four full genotyping assays-the Anyplex (Seegene, Seoul, South Korea) (22), PapilloCheck (Greiner Bio-One, Frickenhausen, Germany) (23), CLART HPV4S (GENOMICA SAU, Madrid, Spain) (24), and Riatol qPCR [Sonic Healthcare Benelux (AML), Antwerp, Belgium] (25)-have been fully validated (13).However, since these assays have not yet achieved the status of standard comparator, they cannot be used in studies to analyze the type-specific performance.
The relative sensitivity of AllplexHPVHR versus RealTime to detect CIN3+ cases was unity.However, the non-inferiority was not statistically significant due to limited number of CIN3+ cases in the current study.The current validation guidelines describe the clinical endpoint of CIN2+.In the future validation guidelines, CIN3+ will be included as an additional clinical endpoint since it is more prone to progress to cancer.It is noteworthy that the role of HPV assay genotyping is assuming an increas ingly important position in CC screening recommendations.Experts concur that the integration of HPV assays with extended genotyping capabilities is essential for more  (26).Such an approach serves the dual purpose of reducing the costs associated with over-screen ing populations and curbing unnecessary referrals.Assays with extended genotyping capabilities offer valuable insights into the changing landscape of HPV genotypes, facilitating the assessment of vaccine impact and informing public health policy decisions.
In conclusion, hrHPV testing on clinician-collected samples using AllplexHPVHR demonstrated non-inferior clinical accuracy to that of the SCT for identifying CIN2+ cases and excellent intra-and inter-laboratory reproducibility according to the interna tional validation criteria.AllplexHPVHR fulfills all the criteria required for the clinical validation of HPV assays and should be considered validated for primary cervical cancer screening.

FIG 1
FIG 1 Meta-analysis of the relative sensitivity for CIN2+ (top) and CIN3+ (middle) and specificity for controls (bottom) of AllplexHPVHR versus the comparator test

TABLE 3
Non-hierarchical and hierarchical ranked genotypes by AllplexHPVHR in the CIN2+ cases

AllplexHPVHR a CIN2+ Non-hierarchical b [n (%)] Hierarchical c [n (%)]
a HPV types are ranked according to their level of carcinogenicity (3).b Non-hierarchical: Samples with all HPV types are counted, irrespective of single or multiple infections.

TABLE 4
Intra-and inter-laboratory reproducibility of AllplexHPVHR

TABLE 5
Genotype-specific level agreement of the AllplexHPVHR assay for intra-laboratory analysis